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---- ----- - Noting that the 1984 presidential campaign was the 

most heavily polled election in United States history, a study 
examined the use and abuse of opinion polls by five print media 
("Newsweek,* "Time," "U.S. News and World Report," the "New York 
Times," and the "Washington Post") and by the three major television 
networks during the campaign. In particular^ the study examined to 
what extent these eight influential media followed sound practices, 
as defined by the American Association for Public Opinion Research 
(AAPOR ) , in the report i ng o f the opinion polls they themselves 
commissioned and over which they exercised control. The AAPOR code 
for reporting public opinion polls includes the following items: (1) 
identity of who sponsored the survey, (2) exact wording of the 
questions asked, (3) definition and size of the population sampled, 

(4) indication of what allowance should be made for sampling error, 

(5) which results are based on part of the sample, (6) whether 
interviewing was done personally, by telephone, by mail, etc., and 
(73-timing of the interviewing. It was found that all the news media 
were deficient in following AAPOR standards in some respects, and 
some were deficient in most respects. While there was no clear winner 
among the three categories of media, network television was a clear 
loser. On every comparison among the three categories of media, the 
networks were more negligent than were the newspapers and news 
magazines. The findings suggest a number of recommendations for 
improved polling performance in the 1988 campaign. (References are 
appended. ) (NKA) 
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ABSTRACT 
Media- Sponsored Opinion Polls 
A Critical Review of Campaign '84 and Recommendations for Campaign '88 

Campaign '84 was the most heavily polled election in this nation's history. This paper examines 
the use and abuse of opinion polls by five prestige print media (Seasflok, lime, U.S. News A 
Wnrlfi RRfmrt , The New York Times, and The Washington PjbO and by the three major TV 
networks during Campaign '84. I n particular , it examines to what extent these eight influential 
media followed sound practices (as defined by the American Association for Public Opinion 
Research) in the reporting of the opinion polls they themselves commissioned and over which 
they exercised control. All of the news media were deficient in following the AAPOR standards in 
some respects, and some of them were deficient in most respects. While there was no clear 
winner among the three categories of media, there was a clear loser: network TV. On every 
comparison between the three categories of media, the networks were more negligert than were 
the newspapers and news magazines. Recommendations are made for improved press 
performance in Campaign '88. 
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Media- Sponsored Opinion Polls. 
A Critical Review of Campaign 84 and Recommendations for Campaign 88 

At the end of the 1 984 presidential campaign, a Ugusttfiel writer ( Alpern, 1 984b:8) 
commented on the use of opinion polls by the mass media: 

Never had a presidential campaign been so bombarded by the political polls, a blizzard of 
statistics- --leads, gaps, strengths and weaknesses- --gushing daily from the 
computers of independent polling firms and the news media. And with profusion came 
confusion 

This paper will examine the use and abuse of opinion polls by five prestige print media 
(HbsssssSl. Iifflfi, U.S. News ft World Report . The New York limsa, end The Washington Essi) 
and by the three major TV networks during Campaign '84 In particular , it will examine to what 
extent these eight influential media followed sound practices in the reporting of the opinion polls 
they themselves commissioned and over which they exercised control. To the extent that there 
were abuses during Campaign "84, what lessons might be learned that could lead to an improved 
use of opinion polls by the media during Campaign "88? 

All toooften.es Wheeler ( 1976 xvi-xvii) has written. "Thereaderof opinion polls is 
given no hint that they may riot be trustworthy. The press in particular has been oblivious to 
the pitfalls of polling." Political scientist David L Palitz and his colleagues ( 1 980:499) stated, 
"The press seems obsessed with presidential elections, willing to publish polls on the subject no 
matter how irrelevant and inane. " 
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The AAPOR Standards 

Normally, when evaluating mass media performance, one of the first problems is 
determining which criteria to use. Against which standards should the media be judged? And 
wto says so? When it comes to evaluating news media reporting of political polls, however , the 
problem is considerably reduced. In 1 968 the American Association for Public Opinion 
Research published Us "Standards for Reporting Public Opinion Polls." The AAPOR code called 
upon the print media and broadcast media to repoi X the following eight pieces of information 
when reporting opinion polls ( Mendelsohn and Crespi , 1 970: 1 34- 1 35): 

1 . identity of who sponsored the survey. 

2. The exact wording of questions asked. 

3. A definition of the population actually sampled. 
4 Size of sample ... 

5. An indication of what allowance should be made for sampling error. 

6. Which results are bas^d on parts of the sample, rather than the total 
sample, for example, likely voters only, those aware of an event, those 
who answered other questions in a certain way, and the like: 

7. Whether interviewing was done personally, by telephone, by mail , or 
on street corners. 

8. Timing of the interviewing 

Some critical observers of polls, however , say that the eight points contained in the 
AAPOR standards are not enough. They believe that the media have an obligation to inform 
readers that polls we subject to adStionai sources of error above and beyond sampling error. 
Wheeler ( 1 976: 1 03) states, for example, that news consumers need to be better informed as to 
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what actually goes on behind the scenes in polling: 

In sum , though the theory of polling is scientifically sound, the actual 
practice is hot ... . Polling is not the pristinely pure scientific process the 
pollsters want us to believe it is ... . But as serious as these technical 
problems are, as suspicious of polls as they should make us, they are hot 
the only source of polling error The fundamental problem of polls lies in 
their analysis. 

And it is this type of warning, says Wheeler, that the hews media usually do not report. 
Pollster Burns W. Roper ( 1980:46) also believes that the news media frequently 

mislead readers, viewers and listeners— not by the facts they report, but by implication: 
Partly because of their lack of deep expertise, reporters have overstressed 
sampling error and undepressed the other more important and 
considerably greater sources or error. And, in the process of stressing 
sampling error , they have not warned the reader or viewer of error as they 
have intended. I nstead they have imp I ied an unwarranted degree of 
accuracy. 

Wheeler ( 1 980 44) states, "It should be at least as important to alert editors- —and 
their readers— to the wide range of possible interpretation as it is to note sampling error. 
Indeed, the standard warning about a 3 or 4 point chance variation doubtlessly misleads some 
readers into believing that the whole report is valid to that degree." 



Applying the Standards 

primary purpose of this critical analysis is straightforward, it is to examine all of 
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the polls sponsored and published by eight prestige news media during Campaign '84 to evaluate 
to what extent they adhered to the AAPOfi standards « The secondary purpose is to analyze the 
news reports to determine what additinrfflMififfflry infnrmntinn- -beyond the points 
Amended by AAPOR— readers were given to help them understand that polling is more art 
than science. 

The time period covered by this critical review wab from August 27, 1 984 ( four days 
after the Republican national convention ended) through November 5. 1 984 ( the day before the 
election). The news magazine poll articles Were located by reading all of the issues p-jbiished 
during the study period. Articles in The New York IMa and The Washington m were located 
with the help of the official index published by each paper The network TV polls Were located by 
*** inQ the Vision N^lrKl P Xf^^trfrh , and video tapes of the network poll stories 
were then obtained from the Vanderbilt Television News Archive: 

The New York lima and CBS, which jointly sponsored some of their polls, were the 
leaders in terms of the total number of poll stories piihl ished ( nine each) The Washington m 
and ABC, which likewise jointly sponsor* some 0 f their polls, were not hit behind, with seven 
and eight poll stories, respectively. Since the news magazines publish only one issue per week , 
one would not expect them to publish as many poll stories as the^ily media, and this indeed was 
the case. The totals for itattfiek, lime andUSjfcas were four, two and one stories, 
respectively. NBC. with its total of two poll stories, gave much less emphasis to poll stories 
than all of the other daily media. 



SPONSOR IDENTIFICATION 

Because of the scope of this analysis, all of the media automatically rated highly in this 
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respect since, in effect, they were oil publishing their own polls. Not only did they identify 
themselves as being the sponsors of the polls, In some cases they also reported the name of the 
polling oration, tkusmt used Gallup, lime used Yankelovich, lLSJte used Civic 
S* vice, Inc. Sponsor identification was simply not a problem for these eight prestige media 
during Campaign '84. 



QUESTION WORDING 

Everyone who has ever taken a true/false or multiple choice exam knows that changing a 
Single word in the question can completely change the answer or results. Thus, even without the 
AAPOR standards, It should be Intuitively obvious to reporters and editors that readers and 
viewers need to know the exact wording of the questions asked in opinion polls if they are to 
meaningfully interpret the results. 

One of the biggest problem areas uncovered in this analysis was that of the nonreporting 
of the verbatim questions asked in the surveys. Of the total of 42 poll stories only six provided 
all of the verbatim questions, and another seven provided seme of the verbatim questions. 
HflSsSsfc and iLSJfcos were the only bright spots in this otherwise bleak picture: In the four 
separate polls published by Mauk-, all (or almost all) of the questions referred to in the 
report were presented verbatim in fact , :heMSS format of grouping the key questions in 
a box , with the questions in regular type and the responses and percentages in boldface type, 
could serve as an excellent model for other print media to follow. 

The one poll published by U^Jte did report the verbetin questions end, like the 
polls, could serve as a model for other print media to follow: JJmg magazine's two 
polls did a generally poor job of reporting verbatim questions; one poll reported some of the key 
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questions and the other reported none. 

The New York limes and The Washington fcosl did worse than the news magazines in 
terms of reporting verbatim questions. The limes provided all of most of the verbatim 
questions in only p » of its nine polls, and reported at le«t me verbatim question in three other 
poll reports. In the five remaining reports, readers were provided with no verbatim questions 
The Washington East provided no verbatin questions in six of its poll stories and some of the 
questions in one story. 

ABB, 6BS and NBC did not carry a single poll story that provided viewers all of the 
relevant verbatim questions. Two of their 19 poll stories provided some of the verbatim 
questions, but the other 1 7 stories totally neglected this information. 

POPULATIONS DEFINED 

Exactly half of the newspaper poll stories (eight out of 16) defined their survey 
populations. The figure for magazines was two out of seven, and the networks provided this 
information in four out of 19 poll stories. The two worst records were those of Lime ( zero out 
Of two) arid 08S ( brie out of eight) Thus, in 28 of the 42 polls the public was given percentages 
relating to Campaign '84 but was riot given an adequate? description of the people the numbers 
were supposed to represent. 

The scores in this area would have been higher if the media would have provided a 
geographic portion of the definition. For example, simply stating "registered voters" was 
counted as not adequately defining the population, but "registered voters nationwide" was 
considers an adequate definition Therefore! while the missing information is fundamental in 
terms of the AAPGR standards, this particular problem would be easy for the media to remedy if 
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they desire to do so. 
SAMPLE SIZES 

All of the print media except the New York JJmfis received perfect scores in their 
reporting of the sample sizes of their surveys, and the omission on the part of the Iims affected 
only >M poll story. The networks neglected this basic Information in six of 1 9 poll stories. The 
worst offender was ABC. which reported the sample size in only half of its eight reports 



INTERVIEWS METHODS 

The news magazines did a good job of reporting their interviewing methods, with the 
single omission being by iiiJ^ The newspapers reported this information in J2of 16 poll 
stories, but the networks once again did poorly. In fact , ABC and NBC rated a zero in providing 
this information to their viewers, and CBS reported the interviewing methods in five out of nine 
stories. 



TIMING 

From the standpoint of reporting the dates of their surveys, all of the print media 
received an almost perfect score The lone exception was one survey published by the New York 
limes Even this exception was a minor one. however . because the story did report the date the 
interviewing fiodfld; ifie reader simply had no way of knowing when the interviewing began. As 
was the case with all of the above comparisons, the performance of the TV networks was not as 
good. Seven of their 1 9 poll stories did not provide complete information about the dates of the 
polling. For example, on October! ABC reported that they had "just finished conducing" a 
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poll. This type of description of timing was not considered acceptable because no indication was 
given concerning the number lays or weeks over which the polling was done. Was it done dur ing 
the last three days or during the last three weeks? This type of information is important in any 
election campaign, where the results of even a single event or debate can guickly influence 
people's opinions. 

SAMPLING ERROR 

Overall, exactly two-thirds of all of the polls reported by these presti.je media included 
specific information about sampling error ( 28 out of 42) The news magazines had the best 
record by including this information in six out of seven poll stories. The performance of the 
newspapers was second best ( 1 1 out of 1 6) and. in keeping with the entire pattern described 
above, the performance of the TV networks was worst ( 1 1 out of 19). In other words, in eight 
of their 1 9 poll stories ( two on ABC, four on CBS. and two on NBC), the viewing audience was 
presented with percentages of how Reagan . Mondale , Bush and Fer arro were doing . but was not 
given any indication of the margins of error . a clear violation ot the AAPOR standards. 

Beyond Sampling Error 
To what extent did these major media voices provide their readers and viewers with 
oiijer. cautionary information beyond the minima recommended by AAPOR? Only one of the news 
magazine stories gave its readers any indication that there are other, possible sources of error 
that ran bias or invalidate a poll, in this one instance, Newsweek's Alpern ( 1 984a:33) did 
imply this by pointing out to readers that a particular sample was ". . a somewhat more 
Reagan-prone audience than the electorate at large . . . ." Likewise, only one network poll story 
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implied this. ABC stated in its October 8th newscast, "Those we polled were more Republican 
and more pro-Reagan than registered voters in general " The Implication of this admission is 
that the sample may have been biased and viewers should take this into consideration. The two 
newspapers did a much better job in this respect, with The New York Iimss providing 
cautionary information in five of its nine polls and The Washington Rosi doing so in three out of 
seven poll reports. 

An interesting finding concerning The New York limes is that . even though it failed to 
provide readers with one of the most basic forms of information, the verbatim questions asked in 
five of its six surveys, it provided other relevant information that rnly a sophisticated poll 
watcher would be likely to appreciate: 

* It reported the Reagan/Mondale preference statistics two ways: with 
"leaning" voters included and with them exglufel 

* It explained that the statistical results were weighted to reflect the 
likllhood of voting. 

* It explained how the telephone exchanges were sampled. 

* It defined "probable electorate " 

* It gave the name and Tcedemie affiliation of its polling consultant; 

* It provided historical context from earlier polls — whether from 
earlier 1 984 polls or from past presidential elections. 

* in one poll that involved Eglnterviews with respondents polled the 
previous week, it explained that the sampling error on the repeated 
questions was less than one percent, while it was four percent on 
the new questions. 
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A similar point can be made with respect to some of the polls published by The 
Washington PjbI Some of the Pjsi for les included 

* Comparative data from earlier East polls for context. 

* Comparative data from other current polls (Gallup, Harris, etc.) for 
context. 

* A statement ( Sussmnn . 1 964:A4) that, "Census Bureau figures for 
age, sex , race and education were used to adjust the sample slightly 
so that it matches the general population in those characteristics." 

* An acknowledgement that the margins of error Tor some smaller 
states were higher than the margins for bigger states: 



Conclusions and Recommendations for Campaign '88 

The major conclusion from this analysis is that all of the news media were deficient in 
following the AAPOR standards in some respects, and some of them were deficient in most 
respects. Out of this conclusion comes a rather obvious recommendation— namely, that these 
influential news sources need to conduct in-service training courses for all of their reporters 
and editors who work on political stories. It's hot enough to have a special polling consultant 
available, even though this In itself is desirable, because with the pressures and short dead! Ines 
that are the norm during a campaign there is simply not enough time to call in the consultant. 
While the front- line reporters and er" tors do not necessarily need to become specialists in the 
development of questionnaires and in sampling theory and methods, they should at least be 
knowledgeable in the AAPOR standards. They need to know sM the standards are and Mft they 
matter. 
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there was no deer winner e mong the three categories of medio, there WQ3 0 c]eQr 
»Osef : TV. even though the number of polls was relatively small arid does not i end 1tself 
to for" 181 Mistical testing, the Pattern that emerges i s quite clear. On every 
co»np fln ' s0n ^tween the three categories of media, the networks wer e more negligent then were 
th^ n^sP^s end hews meg^nes. 

A 11 Observers ana cri ?lcs of netw 0 r k jy news ere aware of the extreme time limitations 
(abot/ 1 25 Chutes of actual time per broadcast) under which they operate. Nevertheless. 
thi s t m M Nation alone dot* m[ ***** for tn e superficiality of much of their poll 
coyer* 5 ' "" " 0ec ause they spend a considerable emount of time reporting various fluff 5(5^ 
about croWds . crowa control , P^ades, slg n $ m(J by people | n the crowdS( ^,3 other interesting 
butn^^tial items 

f urt hermore, it does not take that much time to report most of the essential details. The 
fo1W ing demerit by He«3*^^ Alper n ( 1 98%33). tor example, would take about 2 2 
second t0 re * °n the air end stains six essential points from the AAPOR standards ( sponsor . 
date, ^ etn0dof Polling, sample sjze > population, margin of error): 

For th, s t^tesk Po" • tne 0 81 'up Organization on Sunday night 
telephoned 379 registered voters who watched the debate. They were 
drflWr) from a pool of respondents t n the latest 0a»up poll who said they 
Pitied to watch . and who closely reflected the national sample in 
fcWephjcs, party affiliation and candidate support. The margin of 
error i 3 ~ minu3 7 percentage points. 
Thus jt 588,1,5 t0 oe more a ease of ignorance, rathe r than a lack of time, that accounts for the 
networ* 3 ^^ehcy. in fact, even the fctexs^ statement could be shortened. The word 
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"nationwide'' could be inserted after the word "voters" in the first senteroe, and the second 
sentence, while commendable, could be eliminated. The margin of error could be shown on the 
screen as "Margin of Error: ± 78 N and not have to be read. The slightly revised first sentence 
would then take about eight seconds for the correspondent to red The wording of each question 
could thai either be superimposed on the screen or reed by the correspondent. 

Even though CBS and the other networks were deficient in much of their poll reporting, 
CBS should be praised for a story it carried on October 26th which was not a poll story per se, 
but was a story about election polls in general. This was a long story by TV standards (four and a 
half minutes), and included interviews with various polling experts. It particularly a ddr e ssed 
the question of why different polls come up with different results, and it pointed but that one of 
the reasons is that they use different methods of statistical weighting for likely voters: 

Another criticism of the network TV reporting is that, even when the margins of error 
were shown on the screen, the graphics were so visually cluttered- - -with such things, as 
stars, stripes, donkeys, elephants, the percentages being reported, and so on—that probably 
99 people out of a hundred would not notice the tiny ±3% in the corner of the screen. 
Furthermore, without the words "margin of error" also on the screen, or without the 
correspondent calling attention to the margin of error , presumably very few viewers would get 
the intended message. ABC was the worst offender In terms of hiding the margin of error 
statistic in the cluttered graphics. Since the networks are supposed to be in the business of 
effective communication, all of them should conduct some applied research projects prior to 
Campaign 88 to come up with their own simplified, standardized graphics for reporting poll 
results. Unless 'he networks choose to have the cor respondent read the margin of error 
information, the information should be presented in a manner that is easily seen and read by the 

15 

ERIC 



Media Polls 15 



audience. 

U.S. Itews& World Report deserves to be strongly criticized for its misuse of the terms 
"survey" in four different issues. In its October 8th issue ( Mesne* . 1 984*25) the magazine 
referred to "A U.&^feWs&WariHRflpnri survey conducted in late September . .".and stated 
that this was the "First in a Series of Regional Surveys." in one issue ; Mashek , 1 984b:57) the 
misnomer "Second in a Series of Regional Surveys" in one part of the magazine was especially 
egregious because there was a legitimate survey earlier in the magazine. 

Two recommendations come to mind. First, people who work with words for a living 
should be more careful with the words they use. In particular, journalists should not use the 
terms "survey" and "poll" when they are referring to reporters' informal roundups of people's 
views. Second, MPOR should have a procedure whereby members who spot such abuse should 
immediately notify a designated AAPOR spokesman who would contact the offending media 
organization and request a clarification for readers, listeners or viewers. In the case of U.S. 
News, the abuse went on for four consecutive weeks and was probably not intentional , but 
simply due to ignorance about polling terms. 

E ven if the media did a much better job of reporting the eight pieces of information 
specified by the AAPOR standards, is there a need for additional changes in the way they report 
polls? To this eritic, the answer is a resounding "Yes." The biggest problem that is easily 
correctable is the false impression that even well-reported poll stories give— -namely, that if 
the margin of error is i3S or ±4X, then the results are trustworthy to that extent As several 
pollsters and critics Of polls have pointed out, the margin of error reflects only sampling error. 
"The fundamendal problem of polls lies in their analysis" (Wheeler, 1 976: 1 03). "And in the 
proofs of stressing sampling error, they have hot warned the reader or viewer of error as they 
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have intended. Instead they have impl ied an unwarranted degree of accuracy" ( Roper , 1980 46). 

While it may take years to communicate this point fully to working journalists, there is 
a simple place to begin. AAPOR should revise its standards so that news reports of polls inform 
readers about possible additional sources nf armr Sudman ( 1 983:495) wrote that the media 
should include the following two sentences in their news reports of polls: 
1 These results do not necessarily reflect the views of people who 
refused to be interviewed or could not be found at home during the 
time the survey was done. 
2. Changing the wording or context of some questions might change the 
results; 

To its credit . the New York limes ( Rosenbaum , 1984:89) included the following 
statement in some of its poll stories: "In addition to sampling error, the practical difficulties of 
conducting any survey of public opinion may introduce other sources of error into the poll." 
Likewise, the Washington Eos! (Sussman. I984:A4) should also be praised for including this 
statement in some of its poll stories: "That does not take into account other , nonsampling errors 
that may occur in opinion polling." 

Media-sponsored opinion polls are here to stay, but the deficiencies do not have to be 
here to stay The reporting of opinion polls in Campaign "88 can be significantly better than in 
Campaign "84 ifMPOR members, media executives, and working reporters and editors will 
recognize the past problems and become more informed and involved. 
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Footnote 

1 For the seme reasons stated by Miller and Hurd ( 1 982.245), this analysis did not deal 
with the AAPOR standard of which results are based on parts of the sample, rather than the total 
sample. In addition, it is difficult to evaluate most poll stories on this criterion without having 
access to the original poll data 



20 



